Skip to content

fix(gax-httpjson): use Conscrypt TrustManagerFactory for mTLS SSLContext - #14556

Merged
macastelaz merged 4 commits into
googleapis:agentic-identities-bound-tokenfrom
macastelaz:fix-conscrypt-tmf-13995
Oct 7, 2026
Merged

macastelaz merged 4 commits into
googleapis:agentic-identities-bound-tokenfrom
macastelaz:fix-conscrypt-tmf-13995

Conversation

@macastelaz

@macastelaz macastelaz commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Important

This must merge before the agentic-identities-bound-token feature branch is merged to main. Without it, HTTP/JSON clients fail by default wherever mTLS is enabled automatically (see below).

Problem

When mTLS is active, InstantiatingHttpJsonChannelProvider.configureMtls() builds a Conscrypt SSLContext but initializes it with the JDK (SunJSSE) PKIX TrustManagerFactory. On TLS 1.3, Conscrypt passes authType "GENERIC" to the trust manager. SunJSSE's end-entity checks reject that authType for server certificates that are CA-issued and carry a KeyUsage extension, which includes Google front ends. So every mTLS HTTP/JSON handshake fails with:

javax.net.ssl.SSLHandshakeException: Unknown authType: GENERIC
Caused by: java.security.cert.CertificateException: Unknown authType: GENERIC

The bug is already on main, but there it only triggers when GOOGLE_API_USE_CLIENT_CERTIFICATE=true is set explicitly. On this feature branch, #13995 enables mTLS automatically whenever a workload certificate config is present, so on Cloud Run (agent identity) every HTTP/JSON client fails by default. gRPC isn't affected. The non-mTLS path isn't affected either, because google-http-client already pairs Conscrypt with a provider-matched trust manager there.

A second provider mismatch shows up on JDK 26: the mTLS SSLContext also used the JDK's default (SunX509) KeyManagerFactory. Since JDK 26 (JDK-8359956), that key manager applies algorithm constraints to the client certificate. Called from a Conscrypt handshake, it rejects valid certificates (e.g. SHA256withRSA), so the client sends an empty certificate chain and the server aborts with certificate_required.

Fix

Use Conscrypt's own PKIX TrustManagerFactory (TrustManagerFactory.getInstance("PKIX", conscryptProvider)) for the mTLS SSLContext. I called the JDK API directly rather than SslUtils.getPkixTrustManagerFactory(Provider), which only exists in google-http-client 2.2.0+.

Likewise, use Conscrypt's own KeyManagerFactory (KeyManagerFactory.getInstance("PKIX", conscryptProvider)), so the trust manager and key manager both come from the same provider as the SSLContext.

Conscrypt's trust manager loads the same default trust store as the JDK. Verified on JDK 21: 174 anchors in both, the same set, and both honor -Djavax.net.ssl.trustStore overrides identically.

Testing

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for dynamic mTLS certificate rotation in both gRPC and HTTP/JSON transports. It adds utility methods in MtlsUtils to resolve and validate certificate configurations, implements a CertificateRotationTracker to monitor file fingerprints on disk, and updates ChannelPool and RefreshingHttpJsonChannel to perform thread-safe, hot-swapping of channels upon rotation. Additionally, it updates ApiResultRetryAlgorithm and AttemptCallable to handle UnauthenticatedException by triggering channel refreshes. I have reviewed the changes and identified two areas where error handling and messaging can be improved for robustness.

macastelaz added a commit that referenced this pull request Oct 6, 2026
…ons (#13995)

## Description

This PR adds transparent retries for mTLS workload certificate rotation
to the gRPC and HTTP/JSON transports. When a request fails with
`UNAUTHENTICATED` and the workload certificate on disk has changed, the
transport is rebuilt with the new certificate and the request is retried
once, without interrupting in-flight RPCs or streams.

  ### 🚀 Core Features & Architectural Updates

• Rotation detection: `CertificateRotationTracker` compares the SHA-256
fingerprint of the workload certificate on disk with the certificate the
transport was built with. The check only runs after an auth failure
(never on the request path), and positive results are cached for at most
1 second so a burst of failures doesn't trigger a burst of file reads.
Empty or mid-write files are ignored.
• gRPC: `ChannelPool` replaces all of its channels when a rotation is
detected. Calls already in flight finish on their old channels, which
are shut down once idle. During a rotation refresh, a channel that can't
be recreated is dropped rather than kept, so no traffic keeps using the
old certificate; a statically sized pool is refilled in the background.
• HTTP/JSON: `RefreshingHttpJsonChannel` swaps the channel's
`HttpTransport` for one built with the new certificate. Calls already
created keep using the transport they started with, so in-flight
requests aren't interrupted.
• Retries: `AttemptCallable` (unary) and
`ServerStreamingAttemptCallable` (server streaming) refresh the
transport after an `UNAUTHENTICATED` failure. If the transport moved to
a new certificate during or after the attempt, the failure is flagged
for a single immediate retry on the refreshed channel; its configured
`isRetryable()` value is unchanged.
• `ApiResultRetryAlgorithm` gives that retry once, immediately, without
using a regular attempt.
• If the free retry also fails after another rotation, the call stops
instead of falling back to backoff retries.
• For server streams, the free retry becomes available again once the
stream has made progress, and the stream resumes through the existing
resumption strategy.
• Client and bidi streams refresh the transport on `UNAUTHENTICATED` but
don't retry; the next stream uses the new certificate.
• Plumbing: `TransportChannel` gains `shouldRefresh()`, `refresh()` and
`getGeneration()`, which default to no-ops.
`ApiCallContext.getTransportChannel()` gives retrying callables access
to the channel, and `GrpcCallContext` / `HttpJsonCallContext` carry it
through `merge()` and `withChannel()`.

  ### 🔒 System Hardening & Bug Fixes

• Outstanding RPC leak (`ChannelPool.ReleasingClientCall`): if a call
was cancelled before `start()`, `start()` threw without releasing the
channel entry, leaving the channel with a permanently outstanding RPC
count so it could never be cleaned up after a refresh. The entry is now
released on that path and on cancellation before start.
• HTTP/JSON refresh vs. shutdown: `shutdown()` / `shutdownNow()` are
serialized with `refresh()` so a transport swap can't race with
teardown.

  ### ⚠️ Behavioral & Security Boundary Changes

* mTLS is enabled automatically when a workload certificate config is
present (`MtlsUtils`, auth library):
* When `GOOGLE_API_CERTIFICATE_CONFIG`, or the default gcloud
`certificate_config.json`, contains a `workload` certificate, client
certificates are used even if `GOOGLE_API_USE_CLIENT_CERTIFICATE` is
unset.
      * `GOOGLE_API_USE_CLIENT_CERTIFICATE=false` still turns mTLS off.

  * mTLS misconfiguration now fails closed (`MtlsUtils`):
* Previously, an invalid explicit config was swallowed and the client
silently connected without mTLS.
* Now client creation throws `IllegalStateException` in these cases:
* the explicit `GOOGLE_API_CERTIFICATE_CONFIG` file is missing,
unreadable or malformed;
* the default config file exists but is unreadable or malformed;
          * a configured certificate or key file can't be read.
* When mTLS is required but the transport can't load the client
certificate, gRPC and HTTP/JSON channel creation fails with an
`IOException` instead of falling back to a non-mTLS connection.


### 🧪 Testing
  #### Automated Testing
• Added and updated comprehensive unit-tests reflecting the
thread-safety
fixes inside ChannelPoolTest.java and
RefreshingHttpJsonChannelTest.java.
• Corrected edge case test configurations to leverage realistic mocked
X.509
certificates to properly exercise deep WorkloadCertificateUtils.
  getCertificateFingerprint() filesystem caching mechanisms. 

  #### Manual Testing

End-to-end tests with real generated GAPIC clients against local mTLS
servers, run on this PR's head. Each scenario runs in its own JVM and
checks the exact sequence of client certificates and outcomes the server
saw.

  * **Setup**
* Clients: `KeyManagementServiceClient` over gRPC and HTTP/JSON, and
`GrpcBigQueryReadStub` for server-streaming `ReadRows` with offset-based
resumption.
* Servers: local gRPC and HTTPS servers that require a client
certificate and return `UNAUTHENTICATED` / 401 for "revoked"
certificates (by CN).
* Rotation: the workload certificate under
`GOOGLE_API_CERTIFICATE_CONFIG` is replaced by atomic rename, key first,
then cert.
  * **gRPC unary:**
* basic rotation: one rejected attempt, then a free retry on the new
cert, and no extra attempts afterwards;
      * the server rejects every cert: one free retry, then stop;
* a second rotation during the free retry: stops after 2 attempts, with
no backoff retry;
      * UNAVAILABLE, then rotation;
* corrupt cert on disk: no crash, the call fails, and the client
recovers once a valid cert is written;
* 20-call bursts across 2 rotations less than 1s apart: all succeed with
exactly 1 refresh each;
      * a 3-channel pool: no stale channel after refresh;
* 8 threads across 4 rotations, then `close()` with calls in flight:
70,624/70,624 calls succeeded, no hangs.
  * **gRPC server streaming:**
      * rotation before the stream starts;
* rotation mid-stream: resumes at the right offset with no gaps or
duplicates;
* rotation retry re-armed after the stream makes progress, following an
earlier UNAVAILABLE retry;
* bounded: a second rotation without progress stops the stream, with no
third attempt.
* **HTTP/JSON:** basic rotation; reject-all; double rotation; a slow
call started before rotation completes on the old cert while new calls
use the new cert; corrupt cert, then restore; 8-thread stress with 4
rotations, then `close()` (49,561/49,561 calls succeeded, no hangs). The
socket factory was verified for both Conscrypt and plain JDK TLS.
* **No mTLS** (`GOOGLE_API_USE_CLIENT_CERTIFICATE=false`, or no
certificate config): a single attempt with no client cert and no
refresh, over both gRPC and HTTP/JSON, even when the cert files on disk
change.
  * **Results:**
      * JDK 26 (JDK TLS): all 22 scenarios pass.
* JDK 21 (Conscrypt): all gRPC, streaming and no-mTLS scenarios pass.
* HTTP/JSON with mTLS and Conscrypt on TLS 1.3 fails with an existing
`Unknown authType: GENERIC` handshake error, which #14556 fixes. With
#14556 applied on top of this PR, all 22 scenarios pass on JDK 21 with
Conscrypt active.
      * Harness source: https://paste.googleplex.com/6120902048219136
* Logs: https://paste.googleplex.com/5467233782988800 (this PR on JDK 26
and JDK 21), https://paste.googleplex.com/5563956459077632 (this PR +
#14556 on JDK 21)
@macastelaz
macastelaz force-pushed the fix-conscrypt-tmf-13995 branch from c6fcb75 to bc52d23 Compare October 6, 2026 13:49
@macastelaz macastelaz changed the title fix(gax-httpjson): use Conscrypt TrustManagerFactory for mTLS SSLContext [blocked on #13995] fix(gax-httpjson): use Conscrypt TrustManagerFactory for mTLS SSLContext Oct 6, 2026
@macastelaz
macastelaz marked this pull request as ready for review October 6, 2026 13:53
@macastelaz
macastelaz requested review from a team as code owners October 6, 2026 13:53
@macastelaz
macastelaz requested a review from lqiu96 October 6, 2026 13:54
The mTLS HttpTransport paired a Conscrypt SSLContext with the JDK (SunJSSE)
PKIX TrustManagerFactory. On TLS 1.3 Conscrypt passes authType "GENERIC" to
the trust manager, which SunJSSE rejects for CA-issued server certificates
with a KeyUsage extension (e.g. Google front ends), so every mTLS HTTP/JSON
handshake failed with "Unknown authType: GENERIC". Since this PR enables
mTLS automatically when a workload certificate config is present, this broke
HTTP/JSON clients by default on Cloud Run.

Use Conscrypt's own PKIX TrustManagerFactory, which loads the same default
trust store (including javax.net.ssl.trustStore overrides) as the JDK.

Adds a TLS 1.3 handshake regression test against a local server presenting a
CA-issued leaf with KeyUsage and requiring a client certificate.
Assumptions.assumeTrue requires opentest4j (TestAbortedException), which is
not on the Bazel test classpath, breaking the sdk-platform-java integration
jobs. A class-level @EnabledIf condition has the same skip semantics without
that dependency.
@macastelaz
macastelaz force-pushed the fix-conscrypt-tmf-13995 branch from 7289240 to d54e134 Compare October 6, 2026 17:08
Since JDK 26 (JDK-8359956) the JDK's SunX509 key manager applies algorithm
constraints to the client certificate. When called from a Conscrypt
handshake it rejects valid certificates (e.g. SHA256withRSA), so no client
certificate is sent and mTLS HTTP/JSON handshakes fail. Take the key
manager from Conscrypt, matching the trust manager.
} finally {
response.disconnect();
}
assertThat(clientCertificatePresented.get(10, TimeUnit.SECONDS)).isTrue();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looking at the logic, this doesn't seem like it would take that long. Is this to guard against it running forever? If it does take an unknown amount of time, can we use awaitility here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it's just a fail-safe bound on Future.get() to guard against hanging forever if the server thread stalls. Since .execute() above is synchronous, the server has already completed the handshake and written the response by the time we reach this line, so Future.get() returns immediately without polling (and awaitility isn't currently on the gax-httpjson Maven/Bazel test classpaths). Added a one-line comment above it to clarify.

Comment on lines 237 to +243
SslUtils.initSslContext(
sslContext,
null,
SslUtils.getPkixTrustManagerFactory(),
TrustManagerFactory.getInstance("PKIX", conscryptProvider),
mtlsKeyStore,
"",
SslUtils.getDefaultKeyManagerFactory());
KeyManagerFactory.getInstance("PKIX", conscryptProvider));

@lqiu96 lqiu96 Oct 7, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this fix. Looks like would be ideal if the upstream http-client initialized with PKIX by default, so this could all just be done within builder.trustCertificates. In my mind, we don't need this override (L219-244). But since SslUtils.getDefaultKeyManagerFactory() looks to default with SunX509 and changing that is a behavior breaking change, I think this is the best we can do.

Can you add a small note of the above. e.g. Http-client looks to initialize with the default of SunX509 which is not supported by Conscrypt.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a note in the comment explaining that google-http-client's trustCertificates() initializes with the default SunX509 KeyManagerFactory, which is not supported by Conscrypt.

@macastelaz
macastelaz merged commit cf211ea into googleapis:agentic-identities-bound-token Oct 7, 2026
223 of 260 checks passed
macastelaz added a commit that referenced this pull request Oct 10, 2026
…te-bound ID tokens (#14559)

> [!IMPORTANT]
> **This must merge before the `agentic-identities-bound-token` feature
branch is merged to `main`.** Docs PR #14557 documents this option.

## Why

When an agent identity certificate is available,
`ComputeEngineCredentials` requests certificate-bound ID tokens by
default. A bound ID token is only accepted by targets that authenticate
the caller over mTLS with the same certificate. Targets reached over
standard HTTPS, such as a Cloud Run service's `*.run.app` URL or a
custom domain, reject it with `401 Unauthorized`.

Per the [ID token binding
1-pager](https://docs.google.com/document/d/1yL_xdDimlUlUyid6HDtqZUIMQjX9VznwzXn1SBABjZc/edit?pli=1&resourcekey=0-GxSXGdfH5Ni_nSI9f7xkQg&tab=t.0#heading=h.94fnt3dk2gx)
(Sep 30 decision), ID tokens stay bound by default, and callers get a
per-target `bind_id_token` setting to turn binding off where needed. The
process-wide opt-out (`GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false`) is
unchanged.

## What changes

- New `IdTokenProvider.Option.DISABLE_BOUND_ID_TOKEN`. Callers pass it
per target, the same way as `FORMAT_FULL` and `LICENSES_TRUE`:

  ```java
  IdTokenCredentials.newBuilder()
      .setIdTokenProvider((IdTokenProvider) credentials)
      .setTargetAudience("https://my-service-12345.us-central1.run.app")

.setOptions(Arrays.asList(IdTokenProvider.Option.DISABLE_BOUND_ID_TOKEN))
      .build();
  ```

- `ComputeEngineCredentials.idTokenWithAudience()`: with the option, the
certificate lookup is skipped and the identity endpoint is called with a
plain GET. Without it, behavior is unchanged (bound whenever a
certificate is available and binding isn't turned off by the env var).
- Other credential types (`ServiceAccountCredentials`,
`ImpersonatedCredentials`, `UserCredentials`) already ignore options
they don't use, so the same code works outside agent identity
environments.
- Access tokens are unaffected.

### API shape

`Option` is a set of on/off flags rather than a value, so this adds only
the "off" flag (`DISABLE_BOUND_ID_TOKEN`). Leaving it unset means
"auto", which currently means bound. This matches the 1-pager's
tri-state (unset = auto) without adding an "enable" flag that would do
nothing today; one can be added later without breaking callers if "auto"
starts deciding per target. Python exposes the same control as
`bind_id_token: Optional[bool]` (googleapis/google-cloud-python#18559).

## Tests

### Unit tests

New `ComputeEngineCredentialsTest` cases, all with an agent identity
certificate present and binding enabled:

- `idTokenWithAudience_disableBoundIdToken_requestsUnboundToken`: GET,
no request body, no forced `format=full`.
-
`idTokenWithAudience_disableBoundIdTokenWithFormatFull_requestsUnboundFullToken`:
still GET; `format=full` is kept when the caller asks for it.
- `idTokenWithAudience_disableBoundIdToken_skipsCertificateLookup`: with
a certificate config pointing to a missing file, the default request
fails, but the opted-out request succeeds, which shows the certificate
is never read.
-
`idTokenCredentials_withDisableBoundIdTokenOption_requestsUnboundToken`:
the option set on `IdTokenCredentials` reaches
`ComputeEngineCredentials`.

The existing bound-ID-token tests are unchanged and still pass. Full
`oauth2_http` suite: 1064 tests, 0 failures. `fmt` is clean.

### Live test on GKE

Run on a GKE cluster with agent identity enabled, using a test-only
local merge of this PR with #14595 (GKE support) and #14556 (already on
the base branch). A single process fetches ID tokens through
`IdTokenCredentials` from `ComputeEngineCredentials` (ADC) and sends
them to an agent-to-agent receiver pod. The receiver accepts only bound
tokens over mTLS (whose `cnf` must match the client certificate), and
only unbound tokens over plain HTTP.

| Check | Defaults | `GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false` |
|---|---|---|
| ID token with `DISABLE_BOUND_ID_TOKEN` | **unbound** (no `cnf`);
receiver: plain HTTP 200, mTLS 401 | unbound |
| ID token for another target without the option, same credentials,
fetched afterwards | **still bound** (`cnf` = certificate leaf);
receiver: mTLS 200 | unbound |
| Access token after the ID-token opt-out | **still bound** (401 on a
plain, non-mTLS endpoint) | unbound |

The option unbinds only the ID token it's set on. Other targets and
access tokens are unaffected, and it's harmless when binding is already
off.

**Internal links (Googlers only):**
[results](https://paste.googleplex.com/5926225961418752) ·
[harness](https://paste.googleplex.com/5713819712749568) · cluster
[setup](https://paste.googleplex.com/5635555912712192) and
[manifests](https://paste.googleplex.com/6258309762514944) (shared with
#14595)
macastelaz added a commit that referenced this pull request Oct 14, 2026
…kens and mTLS (#14595)

## Why

GKE delivers agent identity credentials (Pod Certificates) as a single
combined PEM file, holding the certificate chain and a PKCS#8 private
key, at:

```
/var/run/secrets/workload-spiffe-credentials/x509.credential-bundle.private-key.pem
```

The feature branch already handles Cloud Run, where a certificate config
points into the well-known directory. On GKE there is no certificate
config, so neither the transport nor the token side found the
certificate. This is the Java counterpart of
googleapis/google-cloud-python#18559. Background:
go/gke-security-identity/features/agent_identity_diy.md and
go/sdk-mds-bound-token.

## What changes

When **no certificate configuration exists**
(`GOOGLE_API_CERTIFICATE_CONFIG` is unset and there is no
`~/.config/gcloud/certificate_config.json`), a readable, non-empty
bundle now:

- **Transport:** turns on mTLS automatically.
- `MtlsUtils.getWorkloadCertPath` / `useMtlsClientCertificate` return
the bundle path.
- `DefaultMtlsProviderFactory.create()` builds its keystore from the
bundle via a new package-private `X509Provider.createForTransport()`.
- Every GAX transport (`EndpointContext`, gRPC, HTTP/JSON) goes through
that factory.
- **Tokens:** `AgentIdentityUtils` resolves the same bundle, so
`ComputeEngineCredentials` requests tokens bound to the certificate the
transport presents.
- With `GOOGLE_API_USE_CLIENT_CERTIFICATE=true`, the bundle takes
precedence over the Cloud Run well-known files for the same reason.

**Kept unchanged:**

- `GOOGLE_API_USE_CLIENT_CERTIFICATE=false` and
`GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false` still opt out.
- An explicit or default certificate config (including ECP) always beats
the bundle, on both sides.
- **The bundle is never polled for.** GKE writes it before the container
starts, and polling for a missing file would delay every workload that
isn't an agent.
- **Workload identity federation X.509 is unchanged.** The public
`X509Provider` constructors don't fall back to the bundle. Only the
transport factory does.
- If the bundle disappears during the read (rotation), the transport
reports `CertificateSourceUnavailableException` rather than a generic
`IOException`. If it's briefly missing after a credential was cached,
the token side falls back to the cached credential, but only while the
bundle is still allowed (no certificate configuration has appeared).

**New API:** a public static
`MtlsUtils.hasCertificateConfiguration(...)` on an `@InternalApi` class.
This is additive only.

## Differences from Python (intentional)

- Python also checks
`CLOUDSDK_CONTEXT_AWARE_CERTIFICATE_CONFIG_FILE_PATH`. Java has no
equivalent env var.
- Python temporarily keeps default **ID tokens** unbound on Cloud Run
(when `GOOGLE_API_CERTIFICATE_CONFIG` is set). That gate is temporary
and Python-only, so it isn't copied here. On GKE, both libraries bind ID
tokens by default. #14559 adds the per-target `DISABLE_BOUND_ID_TOKEN`
opt-out.

## Related

- #14556 (HTTP/JSON mTLS fix, merged) enables mTLS over HTTP/JSON
transports.
- #14559 (`DISABLE_BOUND_ID_TOKEN`) is the per-target ID-token opt-out.

## Tests

### Unit tests

- `MtlsUtilsTest`:
  - bundle present, absent, a directory, or empty;
  - `USE_CLIENT_CERTIFICATE=false`;
  - a default config without a workload section (ECP) blocks the bundle;
  - an explicit config takes precedence;
  - `hasCertificateConfiguration` / `getGkeCredentialBundlePath`.
- `X509ProviderTest`:
  - transport provider with cert-first and key-first bundles;
  - an explicit config, or an override path, takes precedence;
  - unavailable when there is no bundle;
  - the public constructor ignores the bundle (WIF).
- `AgentIdentityUtilsTest`:
  - implicit discovery with zero sleeps;
  - the cached credential stays bound;
- opt-outs, a default gcloud config, and a non-agent cert all leave the
token unbound;
  - an empty or absent bundle returns null without polling;
- explicit true finds the bundle, never polls for it, and prefers it
over the Cloud Run files;
- fallback to the cache after the bundle is deleted, and no fallback
once a certificate config has appeared (implicit and explicit paths).

Full `oauth2_http` suite: 1089 tests, 0 failures. `fmt` is clean.

### Live test on GKE

- **Cluster:** a GKE cluster with agent identity enabled.
- **Pods:** the test pods get the agent identity credential bundle at
`/var/run/secrets/workload-spiffe-credentials/x509.credential-bundle.private-key.pem`.
- **Image:** built from this PR's jars merged with #14556, which is
already on the base branch and is required for HTTP/JSON mTLS. It uses
ADC with default settings and no client options.
- **Details:** cluster setup, manifests and full logs are in the
internal links below.

**Checks:**
- **GCS over raw HTTP:** read through `storage.mtls.googleapis.com` with
the transport certificate. The same token on plain
`storage.googleapis.com` gets 401, which proves the token is bound.
- **GAX Resource Manager client:** `getProject` over gRPC and HTTP/JSON.
- **ID tokens:** sent to an agent-to-agent receiver pod. It accepts only
bound tokens over mTLS whose `cnf` matches the client certificate, and
only unbound tokens over plain HTTP.

| Scenario | Token | GAX resolved endpoint | GAX gRPC / HTTP-JSON | GCS
mTLS / plain | ID token mTLS / plain |
|---|---|---|---|---|---|
| Defaults | Bound; `cnf` = bundle leaf = transport cert |
`cloudresourcemanager.mtls.googleapis.com` | ✅ / ✅ | 200 / **401** |
**200** / 401 |
| `GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false` | Unbound | mTLS
endpoint | ✅ / ✅ | 200 / 200 | 401 / **200** |
| `GOOGLE_API_USE_CLIENT_CERTIFICATE=false` | Unbound | plain endpoint |
✅ / ✅ | 200 / 200 | 401 / **200** |
| `GOOGLE_API_USE_CLIENT_CERTIFICATE=true` | Bound (bundle) | mTLS
endpoint | ✅ / ✅ | 200 / **401** | **200** / 401 |
| Pod without agent identity | Unbound, no mTLS | plain endpoint |
authenticated / authenticated | — / authenticated | n/a |

**Timing:** the first token fetch took about 0.5–1s in every scenario. A
pod without the bundle has no polling delay.

**Internal links (Googlers only):**
[setup](https://paste.googleplex.com/5635555912712192) ·
[manifests](https://paste.googleplex.com/6258309762514944) ·
[harness](https://paste.googleplex.com/4882197073494016) ·
[build/run](https://paste.googleplex.com/5942061774274560) ·
[results](https://paste.googleplex.com/5072605959290880)

**Out of scope for this PR (tracked separately):**
- **Leaf-only keystore:** `X509Provider`'s keystore holds only the leaf
certificate, through google-http-client
`SecurityUtils.createMtlsKeyStore`. Google APIs over
`*.mtls.googleapis.com` are unaffected. Agent-to-agent mTLS against a
peer that trusts only the SPIFFE root fails unless the caller presents
the full chain.
macastelaz added a commit that referenced this pull request Oct 19, 2026
…r agent identities (#14668)

## Summary

Merges the `agentic-identities-bound-token` feature branch into `main`,
adding end-to-end support for certificate-bound tokens and automatic
mTLS with transparent certificate rotation for agent identities on Cloud
Run and GKE.

### `google-auth-library-java` (`oauth2_http`)

- **Default certificate-bound tokens (`#13873`, `#14595`, `#14658`):**
When an agent identity SPIFFE workload certificate is present (either
via `GOOGLE_API_CERTIFICATE_CONFIG` on Cloud Run or the GKE credential
bundle at
`/var/run/secrets/workload-spiffe-credentials/x509.credential-bundle.private-key.pem`),
`ComputeEngineCredentials` requests certificate-bound access tokens and
ID tokens via `POST` to the Metadata Server with the leaf certificate
chain.
- **Opt-out controls (`#13873`, `#14559`, `#14557`):**
- Process-wide: `GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false` (or legacy
`GOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES=false`)
disables token binding.
- Per-target ID tokens: `IdTokenProvider.Option.DISABLE_BOUND_ID_TOKEN`
requests an unbound ID token for specific targets (such as plain HTTPS
`*.run.app` URLs) while keeping other ID tokens and access tokens bound.
- **Automatic mTLS & strict configuration (`#13995`, `#14595`,
`#14658`):**
- `MtlsUtils.useMtlsClientCertificate()` enables mTLS automatically when
a workload certificate config or GKE credential bundle is present,
unless `GOOGLE_API_USE_CLIENT_CERTIFICATE` is set to a non-`"true"`
value.
- Missing, unreadable, or malformed explicit
`GOOGLE_API_CERTIFICATE_CONFIG` files fail closed with
`IllegalStateException`.
- **Full mTLS certificate chain (`#14621`):** `X509Provider` and
`SecureConnectProvider` now populate the mTLS key store with the full
certificate chain (leaf + intermediates) via
`MtlsKeyStoreUtils.createMtlsKeyStore` rather than only the leaf
certificate.

### `sdk-platform-java/gax-java` (`gax`, `gax-grpc`, `gax-httpjson`)

- **Transparent retries on mTLS certificate rotation (`#13995`):**
- `CertificateRotationTracker` checks the SHA-256 fingerprint of the
workload certificate on disk after an `UNAUTHENTICATED` failure.
- `ChannelPool` (gRPC) and `RefreshingHttpJsonChannel` (HTTP/JSON)
rebuild transport channels with the rotated certificate without
interrupting in-flight RPCs.
- `AttemptCallable` (unary) and `ServerStreamingAttemptCallable` (server
streaming) issue one immediate free retry on the refreshed channel when
a certificate rotation is detected.
- **Fail-closed default mTLS provider loading (`#14658`):** When
`useMtlsClientCertificate()` is `true`, `EndpointContext`,
`InstantiatingGrpcChannelProvider`, and
`InstantiatingHttpJsonChannelProvider` propagate `IOException`s (while
still falling back cleanly on `CertificateSourceUnavailableException`
when no certificate source exists on disk).
- **Conscrypt TLS 1.3 mTLS fix (`#14556`):**
`InstantiatingHttpJsonChannelProvider` uses Conscrypt's own
`TrustManagerFactory` and `KeyManagerFactory` for the mTLS `SSLContext`,
fixing `Unknown authType: GENERIC` on TLS 1.3 and key-manager algorithm
constraint mismatches on JDK 26+.

## Sub-PRs Included on `agentic-identities-bound-token`

| PR | Description |
| :--- | :--- |
| `#13873` | `feat: Enable Bound Token for Agentic Identities` |
| `#13995` | `feat(gax): support transparent retries during mTLS
certificate rotations` |
| `#14556` | `fix(gax-httpjson): use Conscrypt TrustManagerFactory for
mTLS SSLContext` |
| `#14559` | `feat(auth): add DISABLE_BOUND_ID_TOKEN option to opt out
of certificate-bound ID tokens` |
| `#14557` | `docs(auth): document certificate-bound tokens for agent
identities` |
| `#14595` | `feat(auth): support GKE agent identity credential bundle
for bound tokens and mTLS` |
| `#14626` | `fix(gax): drop jsr305 annotations left over after syncing
with main` |
| `#14621` | `fix(auth): present the full certificate chain for mTLS` |
| `#14658` | `fix(auth,gax): align Java mTLS and agent identity
certificate handling with Python parity` |

## Testing

- **Automated tests:** Full unit test suites in
`google-auth-library-java` (`credentials`, `oauth2_http`) and
`sdk-platform-java/gax-java` (`gax`, `gax-grpc`, `gax-httpjson`) pass
across JDK 8, 11, 17, 21, 25, and 26, including new unit tests for
certificate discovery, bound token requests, startup polling, rotation
retries, Conscrypt TLS 1.3 mTLS handshakes, and full certificate chains.
- **Integrated manual testing (`45/45` scenarios passed):** Built all 9
changed artifacts (`google-auth-library-credentials`,
`google-auth-library-oauth2-http`, `api-common`, `gax`, `gax-grpc`,
`gax-httpjson`, `google-cloud-core`, `google-cloud-core-http`,
`google-cloud-core-grpc`) together from the combined branch and ran 22
live GAPIC mTLS rotation scenarios, 14 local bound-token and GAX parity
checks, 3 live Cloud Run Agent Identity job executions, and 6 live GKE
Agent Identity pod scenarios.

<details>
<summary>Full manual test matrix (45 scenarios)</summary>

### 1. Live GAPIC mTLS Rotation (`22/22` Scenarios, `125/125` Checks)

Executed real `KeyManagementServiceClient` (gRPC and HTTP/JSON) and
`BigQueryReadClient` (server-streaming `ReadRows`) against live
`.mtls.googleapis.com` endpoints (`cloudkms.mtls.googleapis.com`,
`bigquerystorage.mtls.googleapis.com`) while rotating client
certificates on disk.

| Group | Scenarios | What Was Verified | Result |
| :--- | :--- | :--- | :--- |
| **Unary gRPC (`G1`–`G7`)** | `atomic-mv`, `inplace`, `symlink-swap`,
`corrupt-recover`, `missing-recover`, `idle-gap`, `burst-25-rpcs` |
`KeyManagementServiceClient` gRPC channel seamlessly picks up rotated
certs across atomic rename, in-place write, Kubernetes-style symlink
swap, corrupt/missing file recovery, idle gaps, and concurrent bursts. |
**7/7 Passed** |
| **Server-Streaming gRPC (`S1`–`S4`)** | `mid-stream`,
`between-streams`, `reconnect-after-rotate`, `corrupt-mid-stream` |
`BigQueryReadClient.readRowsCallable()` active streams continue
uninterrupted during rotation; subsequent streams and reconnects use the
new certificate. | **4/4 Passed** |
| **Unary HTTP/JSON (`H1`–`H5`)** | `atomic-mv`, `inplace`,
`corrupt-recover`, `missing-recover`, `burst-20-rpcs` |
`KeyManagementServiceClient` HTTP/JSON transport reloads rotated certs
and recovers from transient corrupt/missing files without dropping
requests. | **5/5 Passed** |
| **Startup & Providers (`N1a`, `N1b`, `C1`)** | `N1a`, `N1b`,
`C1-conscrypt`, `C1-jdk` | Missing cert config handling at startup; full
rotation works under both Conscrypt and JDK (`SunJSSE`) TLS providers. |
**4/4 Passed** |
| **Combined Assertions** | `G1`–`S4` assertions | Verified `125/125`
individual assertions across 22 isolated JVMs. | **22/22 Passed** |

### 2. Local Bound-Token & GAX Parity (`14/14` Checks)

| Check | PR / Area | What Was Verified | Result |
| :--- | :--- | :--- | :--- |
| **Scenarios 1–7** | `#13873` / `#14595` | Default bound token
(`cnf.x5t#S256` matches leaf),
`GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false`,
`GOOGLE_API_USE_CLIENT_CERTIFICATE=false`, cert rotation updates
`x5t#S256`, and
`GOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES` behavior. |
**Passed** |
| **Scenario 8d** | `#14658` | `GOOGLE_API_USE_CLIENT_CERTIFICATE=true`
with no certs falls back immediately (`43ms` GET) with no 30s polling
delay. | **Passed** |
| **Scenario 8e** | `#14658` | Invalid explicit
`GOOGLE_API_CERTIFICATE_CONFIG` fails closed with `IOException` instead
of silently downgrading. | **Passed** |
| **Scenario 10** | `#14658` | 0-byte startup cert file is polled until
populated rather than failing immediately. | **Passed** |
| **Scenario 11** | `#14658` | Kubernetes `..data` symlinked parent
directory resolves to the real directory for file watching. | **Passed**
|
| **Scenario 12a** | `#14559` |
`IdTokenProvider.Option.DISABLE_BOUND_ID_TOKEN` returns an unbound ID
token without leaking state to subsequent default calls. | **Passed** |
| **Scenario 12b** | `#14621` | `X509Provider` preserves the full
2-certificate chain (`leaf + intermediate`) in the `KeyStore`. |
**Passed** |
| **GAX `fail-closed`** | `#14658` | Missing explicit
`GOOGLE_API_CERTIFICATE_CONFIG` throws `IllegalStateException` in both
`InstantiatingGrpcChannelProvider` and
`InstantiatingHttpJsonChannelProvider`. | **Passed** |
| **GAX `zero-ms-fallback`** | `#14658` |
`GOOGLE_API_USE_CLIENT_CERTIFICATE=true` with no certs builds plain gRPC
and HTTP/JSON channels immediately (`0ms` polling delay). | **Passed** |

### 3. Live Cloud Run Agent Identity (`3/3` Scenarios)

Tested on a Cloud Run Agent Identity job (`--functional-type=agent
--identity-type=agent-identity`).

| Scenario | Env Vars | Key Results |
| :--- | :--- | :--- |
| **Default (Bound)** | *(unset)* | `DefaultMtlsProviderFactory` chain
length = `2` (`#14621`). Access token acquired in `414ms`. ID token
`cnf.x5t#S256` matched leaf cert. `DISABLE_BOUND_ID_TOKEN` (`#14559`)
returned `cnf: false` and next default call restored `cnf: true`. GAX
`ProjectsClient` (gRPC & HTTP/JSON) resolved
`cloudresourcemanager.mtls.googleapis.com:443` and passed mTLS +
bound-token auth (`PERMISSION_DENIED` IAM check). Non-mTLS transport
rejected bound token (`401`), true mTLS passed auth (`403`), and
mismatched client cert was rejected (`401`). |
| **Runtime Bound Opt-Out** |
`GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false` | Unbound access token and
ID token (`cnf` absent). Non-mTLS endpoints passed auth (`403` IAM) and
live Java ADK `LlmAgent` + `Gemini` call on Vertex AI succeeded (`200
OK`). |
| **Client Cert Opt-Out** | `GOOGLE_API_USE_CLIENT_CERTIFICATE=false` |
Unbound access token (`450ms`) and ID token (`cnf` absent). GAX
`ProjectsClient` (gRPC & HTTP/JSON) resolved plain
`cloudresourcemanager.googleapis.com:443` and passed auth
(`PERMISSION_DENIED`). Live Java ADK `LlmAgent` + `Gemini` call on
Vertex AI succeeded (`200 OK`). |

### 4. Live GKE Agent Identity (`6/6` Scenarios)

Tested on a GKE Agent Identity cluster against live GCP APIs
(`cloudresourcemanager`, `storage`) and an in-cluster SPIFFE mTLS
receiver configured to trust **only** the SPIFFE root CA (requiring the
full leaf + intermediate chain from `#14621`).

| Scenario | Pod Annotations | Env Vars | Key Results |
| :--- | :--- | :--- | :--- |
| **`s1-default`** | `inject-podcertificates=true` | *(unset)* | `bundle
exists=true`, first token fetch `1430ms`, SDK KeyStore chain length =
`2`, mTLS receiver (SDK KeyStore) = `200 OK`, mTLS GCS passed auth
(`404`), plain GCS rejected bound token (`401`), gRPC & HTTP/JSON
resolved `.mtls.googleapis.com:443` and passed auth,
`DISABLE_BOUND_ID_TOKEN` -> plain `200` / mTLS `401` / default after
opt-out -> mTLS `200` / access token still bound -> plain GCS `401`. |
| **`s2-runtime-bound-false`** | `inject-podcertificates=true` |
`GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false` | Unbound tokens
(`cnf=null`), mTLS receiver rejected unbound token (`401`), plain
receiver accepted (`200`), raw GCS and Storage SDK (HTTP & gRPC) passed
auth (`404`), gRPC & HTTP/JSON resolved `.mtls.googleapis.com:443` and
passed auth. |
| **`s3-use-cert-false`** | `inject-podcertificates=true` |
`GOOGLE_API_USE_CLIENT_CERTIFICATE=false` | Unbound tokens (`cnf=null`),
gRPC & HTTP/JSON resolved plain
`cloudresourcemanager.googleapis.com:443` and passed auth, plain
receiver = `200`, plain GCS passed auth (`404`). |
| **`s4-use-cert-true`** | `inject-podcertificates=true` |
`GOOGLE_API_USE_CLIENT_CERTIFICATE=true` | Identical bound behavior to
`s1-default`: SDK KeyStore mTLS receiver = `200`,
`.mtls.googleapis.com:443` resolved and passed auth, all
`DISABLE_BOUND_ID_TOKEN` checks passed. |
| **`s5-no-annotation`** | *(none)* | *(unset)* | `bundle exists=false`,
first token fetch `646ms` (no 30s hang), resolved plain
`cloudresourcemanager.googleapis.com:443`, passed auth
(`PERMISSION_DENIED` / `403`). |
| **`s6-use-cert-true-no-annotation`** | *(none)* |
`GOOGLE_API_USE_CLIENT_CERTIFICATE=true` | Verified `#14658` on real
GKE: `bundle exists=false`, first token fetch `609ms` (no 30s polling
delay), resolved plain `cloudresourcemanager.googleapis.com:443`, passed
auth (`PERMISSION_DENIED` / `403`). |

</details>

BEGIN_COMMIT_OVERRIDE
feat(auth): request certificate-bound tokens for agent identities by
default. Set GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false to opt out.
feat(auth): add IdTokenProvider.Option.DISABLE_BOUND_ID_TOKEN to request
unbound ID tokens for specific targets.
feat(auth): support GKE agent identity credential bundle for bound
tokens and mTLS.
fix(auth): present the full certificate chain for mTLS.
docs(auth): known limitation: ADK for Java doesn't support mTLS yet;
agents that use it must set GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false.
feat(gax): enable mTLS automatically when a workload certificate is
present and support transparent retries during mTLS certificate
rotations.
fix(gax-httpjson): use Conscrypt TrustManagerFactory and
KeyManagerFactory for mTLS SSLContext.
END_COMMIT_OVERRIDE

---------

Co-authored-by: Kirill Logachev <kirl@google.com>
Co-authored-by: Min Zhu <zhumin@google.com>
Co-authored-by: Lawrence Qiu <lawrenceqiu@google.com>
Co-authored-by: yangyzs <171981480+yangyzs@users.noreply.github.com>
Co-authored-by: cloud-java-bot <122572305+cloud-java-bot@users.noreply.github.com>
Co-authored-by: Keshav Dandeva <keshavdandeva@google.com>
Co-authored-by: Mattie Fu <mattiefu@google.com>
Co-authored-by: Will Howes <whowes@google.com>
Co-authored-by: Jin Seop Kim <jinseop@google.com>
Co-authored-by: Jiang Zhu <jiangzzhu@google.com>
Co-authored-by: Knut Olav Løite <koloite@gmail.com>
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
Co-authored-by: rahul2393 <irahul@google.com>
Co-authored-by: Sakthivel Subramanian <179120858+sakthivelmanii@users.noreply.github.com>
Co-authored-by: Gaole Meng <gaole@google.com>
Co-authored-by: Pranav Iyer <pjiyer@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants